Module 2: Summarizing and Visualizing Data

library(tidyverse)
demographics = read.csv('data/demos_anonymized.csv')
ids = read.csv('data/ids_anonymized.csv')
model_variables = read.csv('data/model_variables_anonymized.csv')

Introduction

Get a Quick Summary

It’s always a good idea to glance at the data

glimpse(demographics)

summary(demographics[,1:10])

Grouping and Summarizing Operations

A very common operation is to do things by group(s) then create new summary variables.

demographics %>% 
  group_by(libuser) %>% 
  summarise(age = mean(age, na.rm = T))

Grouping and Summarizing Operations

Do multiple operations at once

demographics %>% 
  group_by(libuser) %>% 
  summarise(age_mean = mean(age, na.rm = T),
            age_sd = sd(age, na.rm = T),
            age_max = max(age, na.rm = T),
            prop_male = mean(gender=='Male', na.rm = T))

Mapping functions to groups

We’ll use the map function to map the sum function to each element in the list

x = list(1:3, 4:6, 7:9)
map(x, sum)

Grouping and Summarizing Operations

We can even do this with modeling and other operations.

This extracts the coefficients from a model run for each group.

demographics %>% 
  drop_na(race) %>% 
  group_split(race) %>%   
  map(~lm(award_total_amount ~ gender, data = .))

Visualization

Visualizing data requires that you: - Consider carefully the information you want to display - And then how you want to display it

Tell a story with the data.

And have some fun with it!

ggplot2

The most widely used visualization package in R

Layers

Visualization can be thought of in a layered fashion

Start with the base, then build up

More pipes

ggplot2 uses a + as a pipe to add layers

library(ggplot2)
ggplot(data) +
  geom_point(aes(x = var1, y = var2)) +
  geom_line() +
  theme(plot.caption = element_text(size = 6))

We can pipe to any ggplot as we did before (%>%)

Aesthetics

Aesthetics (aes) map variables to visual properties

Geoms are the geometric units we want to display

Aesthetics

model_variables %>% 
  filter(award_total_amount > 1e7) %>% 
  ggplot() +
  geom_density(aes(x = award_total_amount, 
                   color = factor(gender),
                   fill = factor(gender)),
               alpha = .2) + 
  scale_x_continuous(breaks = (1:10) * 1e7, trans = 'log')

demographics %>% 
  filter(award_year_start < 2020 & award_year_start > 1990) %>%  
  ggplot(aes(award_year_start, award_total_log)) + 
  geom_smooth(aes(color=factor(libuser)))

Stats

We can also use ggplot2 to create statistics we want to visualize

Typically used indirectly when geoms are called

Can be used for more direct control

ggplot(model_variables, aes(age, award_total_amount)) +
  geom_point(alpha = .02) +
  stat_ellipse(color = '#ff5500')

Scales

Scales are used to add specifications to axes, colors, etc.

model_variables %>% 
  filter(award_total_amount >= 1e6) %>% 
  ggplot() +
  geom_density(aes(x = award_total_amount, 
                   color = gender,
                   fill = gender),
               alpha = .2) + 
  scale_x_continuous(breaks = c(1e6, 5e6, 1e7, 2.5e7, 5e7), 
                     trans = 'log') +
  scale_fill_viridis_d(begin = .25, end = .5) +
  scale_color_viridis_d(begin = .25, end = .5)

Facets

Facets allow another dimension to plots by group

facet_grid: returns a matrix of like dimensions
facet_wrap: more flexible specification

model_variables %>% 
  ggplot() +
  geom_density(aes(x = award_total_amount, 
                   color = libuser,
                   fill = libuser),
               alpha = .2) +
  facet_wrap(~gender)

Themes

Themes allow for customization

Two uses of a - a built-in versions (e.g. theme_minimal) - DIY (theme(…))

For the theme function, each argument, takes on a specific value or an element function:

element_rect
element_line
element_text
element_blank

Themes

model_variables %>% 
  ggplot() +
  geom_smooth(aes(x = age,
                y = award_total_amount,
                color = libuser),
               alpha = .2) + 
  theme_minimal()

model_variables %>% 
  ggplot() +
  geom_smooth(aes(x = age,
                y = award_total_amount,
                color = libuser),
               alpha = .2) + 
  theme(axis.text.x = element_text(size=12),
        panel.grid.minor.x = element_blank(),
        plot.background = element_rect(color = 'rosybrown'),
        panel.background = element_rect(fill = 'papayawhip'))

Interactivity

Interactivity is a must-have tool for web-based presentation

Use to enhance exploration of the data - Not just because one can

Allows for additional dimensions

Even useful for exploring raw data

Interactivity

General

plotly
- also used in Python, Matlab, Julia
- can convert ggplot2 images to interactive ones
highcharter
- general wrapper for highcharts.js
- works with some R packages out of the box
rbokeh
- like plotly, it also has cross language support

Interactivity

Specific functionality:

DT
- interactive data tables
leaflet
- maps with OpenStreetMap
visNetwork
- Network visualization

Plotly

traces - add_, work similar to geoms

modes - allow for points, lines, text and combinations

aesthetics - variables are denoted with ~, constants do not use - x =~ var1 vs x = 2

Plotly

Plotly uses the standard pipe %>%

library(plotly)

model_variables %>% 
  plot_ly(x = ~gender, y = ~ age) %>% 
  add_boxplot(color =~ gender)

Plotly

library(plotly)

model_variables %>% 
  group_by(libuser, gender) %>% 
  summarise(award = mean(award_total_amount)) %>% 
  plot_ly(x = ~libuser, 
          y = ~award, 
          color = ~gender,
          text = ~round(award), 
          textposition = 'auto', 
          type = 'bar') %>% 
  layout(bargap = 0.25, 
         bargroupgap = 0.25)

Plotly

init = glm(award_total_amount >= 5000000 ~ age*libuser, 
           data = model_variables, 
           family = binomial)

model_variables %>% 
  modelr::add_predictions(init, type = 'response') %>% 
  plot_ly(x = ~age, y = ~ pred) %>% 
  add_lines(color =~ libuser, line = list(shape = "spline")) %>% 
  layout(title = 'Predicted Prob. Award > 1 mil')

Plotly

Use ggplotly to turn our formerly static plots into interactive ones.

p = model_variables %>% 
  ggplot() +
  geom_density(aes(x = log(award_total_amount), 
                   color = libuser,
                   fill = libuser),
               alpha = .2) + 
  facet_wrap(~ gender) 
ggplotly()

Python examples

Init

import pandas as pd
import numpy as np

demographics = pd.read_csv('data/demos_anonymized.csv')
ids = pd.read_csv('data/ids_anonymized.csv')
model_variables = pd.read_csv('data/model_variables_anonymized.csv')

Grouping and Summarizing data

demographics.describe(include = 'all')

demographics.describe(include = [np.number])

demographics.describe(include = [np.object])

lib_group = demographics.groupby('libuser', sort=True, )

# automatically chooses numeric
lib_group.mean()

lib_group.get_group(0).head()

lib_group.size()

lib_group.describe()

Mapping a function

x = [[1,2,3], [4,5,6], [7,8,9]]
list(map(np.sum, x))

demographics.select_dtypes('number').apply(np.mean)

Visualization

matplotlib is the most common visualization module in Python, though it’s fairly dated at this point. As such we’ll use a ggplot implementation in python called plotnine.

Unfortunately for plotly, the interactivity makes it unusable within the R notebook (at present), so you may need to switch to Anaconda or other IDE to try other modules like plotly. Even Python users will still use R for easier visualization though, so feel free to do what you like there, then use ggplot etc. in R when the time comes.

That said, I’ll show a couple plots

import plotnine

dplot = demographics[(demographics.award_year_start < 2020) & (demographics.award_year_start > 1990) & (demographics.award_total_log > 11)]
dplot
ggplot(dplot, aes(x='award_year_start', y='award_total_log')) + geom_smooth(aes(group = 'libuser', color = 'libuser'))

year_award_average = demographics[demographics.libuser == 1].groupby(['award_home_dept', 'award_year_start']).mean()
year_award_average

import plotly
import plotly.plotly as py
from plotly.offline import init_notebook_mode
import plotly.graph_objs as go
plotly.offline.init_notebook_mode(connected=True)

y0 = np.random.randn(50)-1
y1 = np.random.randn(50)+1

trace0 = go.Box(
    y=y0
)
trace1 = go.Box(
    y=y1
)
data = [trace0, trace1]
py.iplot(data)

LS0tCnRpdGxlOiAiTW9kdWxlIDI6IFN1bW1hcml6aW5nIGFuZCBWaXN1YWxpemluZyBEYXRhIgpvdXRwdXQ6IAogIGh0bWxfbm90ZWJvb2s6IAogICAgaGlnaGxpZ2h0OiBweWdtZW50cwogICAgdGhlbWU6IHNhbmRzdG9uZQogICAgY3NzOiBvdGhlci5jc3MKZWRpdG9yX29wdGlvbnM6IAogIGNodW5rX291dHB1dF90eXBlOiBpbmxpbmUKLS0tCgpgYGB7ciBpbml0LCBlY2hvPUZBTFNFfQojIHRoZXNlIG9wdGlvbnMgYXJlIHByaW1hcnkgdXNlZnVsIHRvIHRoZSBjcmVhdGlvbiBvZiB0aGUgaHRtbCBkb2N1bWVudAprbml0cjo6b3B0c19jaHVuayRzZXQoCiAgZWNobz1ULCAKICBldmFsID0gRiwKICBtZXNzYWdlID0gRiwgCiAgd2FybmluZyA9IEYsIAogIGNvbW1lbnQgPSBOQSwKICBSLm9wdGlvbnM9bGlzdCh3aWR0aD0xMjApLCAKICBjYWNoZS5yZWJ1aWxkPUYsIAogIGNhY2hlPUYsCiAgZmlnLmFsaWduPSdjZW50ZXInLCAKICBmaWcuYXNwID0gLjcsCiAgZGV2ID0gJ3N2ZycsIAogIGRldi5hcmdzPWxpc3QoYmcgPSAndHJhbnNwYXJlbnQnKQopCmBgYAoKYGBge3IgY2F0Y2h1cCwgZXZhbD1UUlVFfQpsaWJyYXJ5KHRpZHl2ZXJzZSkKZGVtb2dyYXBoaWNzID0gcmVhZC5jc3YoJ2RhdGEvZGVtb3NfYW5vbnltaXplZC5jc3YnKQppZHMgPSByZWFkLmNzdignZGF0YS9pZHNfYW5vbnltaXplZC5jc3YnKQptb2RlbF92YXJpYWJsZXMgPSByZWFkLmNzdignZGF0YS9tb2RlbF92YXJpYWJsZXNfYW5vbnltaXplZC5jc3YnKQpgYGAKCiMjIEludHJvZHVjdGlvbgoKIyMgR2V0IGEgUXVpY2sgU3VtbWFyeQoKSXQncyBhbHdheXMgYSBnb29kIGlkZWEgdG8gZ2xhbmNlIGF0IHRoZSBkYXRhCgpgYGB7ciBzdW1tYXJ5fQpnbGltcHNlKGRlbW9ncmFwaGljcykKYGBgCmBgYHtyfQpzdW1tYXJ5KGRlbW9ncmFwaGljc1ssMToxMF0pCmBgYAoKCgojIyBHcm91cGluZyBhbmQgU3VtbWFyaXppbmcgT3BlcmF0aW9ucwoKQSB2ZXJ5IGNvbW1vbiBvcGVyYXRpb24gaXMgdG8gZG8gdGhpbmdzIGJ5IGdyb3VwKHMpIHRoZW4gY3JlYXRlIG5ldyBzdW1tYXJ5IHZhcmlhYmxlcy4KCmBgYHtyIGdyb3VwX2J5LCBldmFsPVRSVUV9CmRlbW9ncmFwaGljcyAlPiUgCiAgZ3JvdXBfYnkobGlidXNlcikgJT4lIAogIHN1bW1hcmlzZShhZ2UgPSBtZWFuKGFnZSwgbmEucm0gPSBUKSkgCmBgYAoKCgojIyBHcm91cGluZyBhbmQgU3VtbWFyaXppbmcgT3BlcmF0aW9ucwoKRG8gbXVsdGlwbGUgb3BlcmF0aW9ucyBhdCBvbmNlCgpgYGB7ciBncm91cF9ieTIsIGV2YWw9VFJVRX0KZGVtb2dyYXBoaWNzICU+JSAKICBncm91cF9ieShsaWJ1c2VyKSAlPiUgCiAgc3VtbWFyaXNlKGFnZV9tZWFuID0gbWVhbihhZ2UsIG5hLnJtID0gVCksCiAgICAgICAgICAgIGFnZV9zZCA9IHNkKGFnZSwgbmEucm0gPSBUKSwKICAgICAgICAgICAgYWdlX21heCA9IG1heChhZ2UsIG5hLnJtID0gVCksCiAgICAgICAgICAgIHByb3BfbWFsZSA9IG1lYW4oZ2VuZGVyPT0nTWFsZScsIG5hLnJtID0gVCkpIApgYGAKCgoKIyMgIE1hcHBpbmcgZnVuY3Rpb25zIHRvIGdyb3VwcwoKV2XigJlsbCB1c2UgdGhlIGBtYXBgIGZ1bmN0aW9uIHRvICptYXAqIHRoZSBgc3VtYCBmdW5jdGlvbiB0byBlYWNoIGVsZW1lbnQgaW4gdGhlIGxpc3QKCmBgYHtyIG1hcCwgZXZhbD1UUlVFfQp4ID0gbGlzdCgxOjMsIDQ6NiwgNzo5KQptYXAoeCwgc3VtKQpgYGAKCgoKIyMgR3JvdXBpbmcgYW5kIFN1bW1hcml6aW5nIE9wZXJhdGlvbnMKCldlIGNhbiBldmVuIGRvIHRoaXMgd2l0aCBtb2RlbGluZyBhbmQgb3RoZXIgb3BlcmF0aW9ucy4gCgpUaGlzIGV4dHJhY3RzIHRoZSBjb2VmZmljaWVudHMgZnJvbSBhIG1vZGVsIHJ1biBmb3IgZWFjaCBncm91cC4KCmBgYHtyIGdyb3VwX2J5MywgZXZhbD1UUlVFfQpkZW1vZ3JhcGhpY3MgJT4lIAogIGRyb3BfbmEocmFjZSkgJT4lIAogIGdyb3VwX3NwbGl0KHJhY2UpICU+JSAgIAogIG1hcCh+bG0oYXdhcmRfdG90YWxfYW1vdW50IH4gZ2VuZGVyLCBkYXRhID0gLikpCmBgYAoKCiMjIFZpc3VhbGl6YXRpb24KClZpc3VhbGl6aW5nIGRhdGEgcmVxdWlyZXMgdGhhdCB5b3U6Ci0gQ29uc2lkZXIgY2FyZWZ1bGx5IHRoZSBpbmZvcm1hdGlvbiB5b3Ugd2FudCB0byBkaXNwbGF5Ci0gQW5kIHRoZW4gaG93IHlvdSB3YW50IHRvIGRpc3BsYXkgaXQKClRlbGwgYSBzdG9yeSB3aXRoIHRoZSBkYXRhLgoKQW5kIGhhdmUgc29tZSBmdW4gd2l0aCBpdCEKCgojIyBnZ3Bsb3QyCgpUaGUgbW9zdCB3aWRlbHkgdXNlZCB2aXN1YWxpemF0aW9uIHBhY2thZ2UgaW4gUgoKCgojIyBMYXllcnMKClZpc3VhbGl6YXRpb24gY2FuIGJlIHRob3VnaHQgb2YgaW4gYSBsYXllcmVkIGZhc2hpb24KClN0YXJ0IHdpdGggdGhlIGJhc2UsIHRoZW4gYnVpbGQgdXAKCiMjIE1vcmUgcGlwZXMKCmBnZ3Bsb3QyYCB1c2VzIGEgYCtgIGFzIGEgcGlwZSB0byBhZGQgbGF5ZXJzCgpgYGB7ciBnZ3Bsb3QyLCBldmFsPUZBTFNFfQpsaWJyYXJ5KGdncGxvdDIpCmdncGxvdChkYXRhKSArCiAgZ2VvbV9wb2ludChhZXMoeCA9IHZhcjEsIHkgPSB2YXIyKSkgKwogIGdlb21fbGluZSgpICsKICB0aGVtZShwbG90LmNhcHRpb24gPSBlbGVtZW50X3RleHQoc2l6ZSA9IDYpKQpgYGAKCldlIGNhbiBwaXBlICp0byogYW55IGdncGxvdCBhcyB3ZSBkaWQgYmVmb3JlIChgICU+JSBgKQoKCgojIyBBZXN0aGV0aWNzCgoqKkFlc3RoZXRpY3MqKiAoYGFlc2ApIG1hcCB2YXJpYWJsZXMgdG8gdmlzdWFsIHByb3BlcnRpZXMKCioqR2VvbXMqKiBhcmUgdGhlIGdlb21ldHJpYyB1bml0cyB3ZSB3YW50IHRvIGRpc3BsYXkKCgojIyBBZXN0aGV0aWNzCgpgYGB7ciBhZXN0aGV0aWNzfQptb2RlbF92YXJpYWJsZXMgJT4lIAogIGZpbHRlcihhd2FyZF90b3RhbF9hbW91bnQgPiAxZTcpICU+JSAKICBnZ3Bsb3QoKSArCiAgZ2VvbV9kZW5zaXR5KGFlcyh4ID0gYXdhcmRfdG90YWxfYW1vdW50LCAKICAgICAgICAgICAgICAgICAgIGNvbG9yID0gZmFjdG9yKGdlbmRlciksCiAgICAgICAgICAgICAgICAgICBmaWxsID0gZmFjdG9yKGdlbmRlcikpLAogICAgICAgICAgICAgICBhbHBoYSA9IC4yKSArIAogIHNjYWxlX3hfY29udGludW91cyhicmVha3MgPSAoMToxMCkgKiAxZTcsIHRyYW5zID0gJ2xvZycpCmBgYAoKYGBge3IgYWVzdGhldGljczJ9CmRlbW9ncmFwaGljcyAlPiUgCiAgZmlsdGVyKGF3YXJkX3llYXJfc3RhcnQgPCAyMDIwICYgYXdhcmRfeWVhcl9zdGFydCA+IDE5OTApICU+JSAgCiAgZ2dwbG90KGFlcyhhd2FyZF95ZWFyX3N0YXJ0LCBhd2FyZF90b3RhbF9sb2cpKSArIAogIGdlb21fc21vb3RoKGFlcyhjb2xvcj1mYWN0b3IobGlidXNlcikpKQpgYGAKCgojIyBTdGF0cwoKV2UgY2FuIGFsc28gdXNlIGdncGxvdDIgdG8gY3JlYXRlIHN0YXRpc3RpY3Mgd2Ugd2FudCB0byB2aXN1YWxpemUKClR5cGljYWxseSB1c2VkIGluZGlyZWN0bHkgd2hlbiBnZW9tcyBhcmUgY2FsbGVkCgpDYW4gYmUgdXNlZCBmb3IgbW9yZSBkaXJlY3QgY29udHJvbAoKYGBge3IgZ2dzdGF0czJ9CmdncGxvdChtb2RlbF92YXJpYWJsZXMsIGFlcyhhZ2UsIGF3YXJkX3RvdGFsX2Ftb3VudCkpICsKICBnZW9tX3BvaW50KGFscGhhID0gLjAyKSArCiAgc3RhdF9lbGxpcHNlKGNvbG9yID0gJyNmZjU1MDAnKQpgYGAKCgoKIyMgU2NhbGVzCgpTY2FsZXMgYXJlIHVzZWQgdG8gYWRkIHNwZWNpZmljYXRpb25zIHRvIGF4ZXMsIGNvbG9ycywgZXRjLgoKYGBge3Igc2NhbGVzfQptb2RlbF92YXJpYWJsZXMgJT4lIAogIGZpbHRlcihhd2FyZF90b3RhbF9hbW91bnQgPj0gMWU2KSAlPiUgCiAgZ2dwbG90KCkgKwogIGdlb21fZGVuc2l0eShhZXMoeCA9IGF3YXJkX3RvdGFsX2Ftb3VudCwgCiAgICAgICAgICAgICAgICAgICBjb2xvciA9IGdlbmRlciwKICAgICAgICAgICAgICAgICAgIGZpbGwgPSBnZW5kZXIpLAogICAgICAgICAgICAgICBhbHBoYSA9IC4yKSArIAogIHNjYWxlX3hfY29udGludW91cyhicmVha3MgPSBjKDFlNiwgNWU2LCAxZTcsIDIuNWU3LCA1ZTcpLCAKICAgICAgICAgICAgICAgICAgICAgdHJhbnMgPSAnbG9nJykgKwogIHNjYWxlX2ZpbGxfdmlyaWRpc19kKGJlZ2luID0gLjI1LCBlbmQgPSAuNSkgKwogIHNjYWxlX2NvbG9yX3ZpcmlkaXNfZChiZWdpbiA9IC4yNSwgZW5kID0gLjUpIApgYGAKCgojIyBGYWNldHMKCkZhY2V0cyBhbGxvdyBhbm90aGVyIGRpbWVuc2lvbiB0byBwbG90cyBieSBncm91cAoKLSAqKmZhY2V0X2dyaWQqKjogcmV0dXJucyBhIG1hdHJpeCBvZiBsaWtlIGRpbWVuc2lvbnMKLSAqKmZhY2V0X3dyYXAqKjogbW9yZSBmbGV4aWJsZSBzcGVjaWZpY2F0aW9uCgpgYGB7ciBmYWNldHN9Cm1vZGVsX3ZhcmlhYmxlcyAlPiUgCiAgZ2dwbG90KCkgKwogIGdlb21fZGVuc2l0eShhZXMoeCA9IGF3YXJkX3RvdGFsX2Ftb3VudCwgCiAgICAgICAgICAgICAgICAgICBjb2xvciA9IGxpYnVzZXIsCiAgICAgICAgICAgICAgICAgICBmaWxsID0gbGlidXNlciksCiAgICAgICAgICAgICAgIGFscGhhID0gLjIpICsKICBmYWNldF93cmFwKH5nZW5kZXIpCmBgYAoKCgoKIyMgVGhlbWVzCgpgVGhlbWVzYCBhbGxvdyBmb3IgY3VzdG9taXphdGlvbgoKVHdvIHVzZXMgb2YgYSAKLSBhIGJ1aWx0LWluIHZlcnNpb25zIChlLmcuIGB0aGVtZV9taW5pbWFsYCkKLSBESVkgIChgdGhlbWVgKC4uLikpCgpGb3IgdGhlIHRoZW1lIGZ1bmN0aW9uLCBlYWNoIGFyZ3VtZW50LCB0YWtlcyBvbiBhIHNwZWNpZmljIHZhbHVlIG9yIGFuIGVsZW1lbnQgZnVuY3Rpb246CgotIGBlbGVtZW50X3JlY3RgCi0gYGVsZW1lbnRfbGluZWAKLSBgZWxlbWVudF90ZXh0YAotIGBlbGVtZW50X2JsYW5rYAoKCiMjIFRoZW1lcwoKYGBge3IgdGhlbWVzMX0KbW9kZWxfdmFyaWFibGVzICU+JSAKICBnZ3Bsb3QoKSArCiAgZ2VvbV9zbW9vdGgoYWVzKHggPSBhZ2UsCiAgICAgICAgICAgICAgICB5ID0gYXdhcmRfdG90YWxfYW1vdW50LAogICAgICAgICAgICAgICAgY29sb3IgPSBsaWJ1c2VyKSwKICAgICAgICAgICAgICAgYWxwaGEgPSAuMikgKyAKICB0aGVtZV9taW5pbWFsKCkKYGBgCgoKYGBge3IgdGhlbWVzMn0KbW9kZWxfdmFyaWFibGVzICU+JSAKICBnZ3Bsb3QoKSArCiAgZ2VvbV9zbW9vdGgoYWVzKHggPSBhZ2UsCiAgICAgICAgICAgICAgICB5ID0gYXdhcmRfdG90YWxfYW1vdW50LAogICAgICAgICAgICAgICAgY29sb3IgPSBsaWJ1c2VyKSwKICAgICAgICAgICAgICAgYWxwaGEgPSAuMikgKyAKICB0aGVtZShheGlzLnRleHQueCA9IGVsZW1lbnRfdGV4dChzaXplPTEyKSwKICAgICAgICBwYW5lbC5ncmlkLm1pbm9yLnggPSBlbGVtZW50X2JsYW5rKCksCiAgICAgICAgcGxvdC5iYWNrZ3JvdW5kID0gZWxlbWVudF9yZWN0KGNvbG9yID0gJ3Jvc3licm93bicpLAogICAgICAgIHBhbmVsLmJhY2tncm91bmQgPSBlbGVtZW50X3JlY3QoZmlsbCA9ICdwYXBheWF3aGlwJykpCmBgYAoKIyMgSW50ZXJhY3Rpdml0eQoKSW50ZXJhY3Rpdml0eSBpcyBhIG11c3QtaGF2ZSB0b29sIGZvciB3ZWItYmFzZWQgcHJlc2VudGF0aW9uCgpVc2UgdG8gZW5oYW5jZSBleHBsb3JhdGlvbiBvZiB0aGUgZGF0YQotIE5vdCBqdXN0IGJlY2F1c2Ugb25lIGNhbgoKQWxsb3dzIGZvciBhZGRpdGlvbmFsIGRpbWVuc2lvbnMKCkV2ZW4gdXNlZnVsIGZvciBleHBsb3JpbmcgcmF3IGRhdGEKCgoKIyMgSW50ZXJhY3Rpdml0eQoKCkdlbmVyYWwgCgotIFtwbG90bHldKGh0dHBzOi8vcGxvdC5seS9yLykKICAtIGFsc28gdXNlZCBpbiBQeXRob24sIE1hdGxhYiwgSnVsaWEKICAtIGNhbiBjb252ZXJ0IGBnZ3Bsb3QyYCBpbWFnZXMgdG8gaW50ZXJhY3RpdmUgb25lcwogIAotIFtoaWdoY2hhcnRlcl0oaHR0cDovL2prdW5zdC5jb20vaGlnaGNoYXJ0ZXIvKQogIC0gZ2VuZXJhbCB3cmFwcGVyIGZvciBoaWdoY2hhcnRzLmpzIAogIC0gd29ya3Mgd2l0aCBzb21lIFIgcGFja2FnZXMgb3V0IG9mIHRoZSBib3gKICAKLSBbcmJva2VoXShodHRwOi8vaGFmZW4uZ2l0aHViLmlvL3Jib2tlaC8pCiAgLSBsaWtlIHBsb3RseSwgaXQgYWxzbyBoYXMgY3Jvc3MgbGFuZ3VhZ2Ugc3VwcG9ydAoKCgojIyBJbnRlcmFjdGl2aXR5CgpTcGVjaWZpYyBmdW5jdGlvbmFsaXR5OgoKLSBbRFRdKGh0dHBzOi8vcnN0dWRpby5naXRodWIuaW8vRFQvKQogIC0gaW50ZXJhY3RpdmUgZGF0YSB0YWJsZXMKICAKLSBbbGVhZmxldF0oaHR0cHM6Ly9yc3R1ZGlvLmdpdGh1Yi5pby9sZWFmbGV0LykKICAgIC0gbWFwcyB3aXRoIE9wZW5TdHJlZXRNYXAKICAgIAotIFt2aXNOZXR3b3JrXShodHRwOi8vZGF0YXN0b3JtLW9wZW4uZ2l0aHViLmlvL3Zpc05ldHdvcmsvKQogICAgLSBOZXR3b3JrIHZpc3VhbGl6YXRpb24KICAgIAoKCiMjIFBsb3RseQoKYHRyYWNlc2AKLSBgYWRkX2AsIHdvcmsgc2ltaWxhciB0byBnZW9tcwogIApgbW9kZXNgCi0gYWxsb3cgZm9yIHBvaW50cywgbGluZXMsIHRleHQgYW5kIGNvbWJpbmF0aW9ucwoKYGFlc3RoZXRpY3NgCi0gdmFyaWFibGVzIGFyZSBkZW5vdGVkIHdpdGggYH5gLCBjb25zdGFudHMgZG8gbm90IHVzZQotIGB4ID1+IHZhcjFgIHZzIGB4ID0gMmAKCgoKCiMjIFBsb3RseQoKUGxvdGx5IHVzZXMgdGhlIHN0YW5kYXJkIHBpcGUgYCU+JWAKCmBgYHtyIHBsb3RseX0KbGlicmFyeShwbG90bHkpCgptb2RlbF92YXJpYWJsZXMgJT4lIAogIHBsb3RfbHkoeCA9IH5nZW5kZXIsIHkgPSB+IGFnZSkgJT4lIAogIGFkZF9ib3hwbG90KGNvbG9yID1+IGdlbmRlcikKYGBgCgoKCiMjIFBsb3RseQoKYGBge3IgcGxvdGx5MX0KbGlicmFyeShwbG90bHkpCgptb2RlbF92YXJpYWJsZXMgJT4lIAogIGdyb3VwX2J5KGxpYnVzZXIsIGdlbmRlcikgJT4lIAogIHN1bW1hcmlzZShhd2FyZCA9IG1lYW4oYXdhcmRfdG90YWxfYW1vdW50KSkgJT4lIAogIHBsb3RfbHkoeCA9IH5saWJ1c2VyLCAKICAgICAgICAgIHkgPSB+YXdhcmQsIAogICAgICAgICAgY29sb3IgPSB+Z2VuZGVyLAogICAgICAgICAgdGV4dCA9IH5yb3VuZChhd2FyZCksIAogICAgICAgICAgdGV4dHBvc2l0aW9uID0gJ2F1dG8nLCAKICAgICAgICAgIHR5cGUgPSAnYmFyJykgJT4lIAogIGxheW91dChiYXJnYXAgPSAwLjI1LCAKICAgICAgICAgYmFyZ3JvdXBnYXAgPSAwLjI1KQpgYGAKCgoKIyMgUGxvdGx5CgpgYGB7ciBwbG90bHkyfQppbml0ID0gZ2xtKGF3YXJkX3RvdGFsX2Ftb3VudCA+PSA1MDAwMDAwIH4gYWdlKmxpYnVzZXIsIAogICAgICAgICAgIGRhdGEgPSBtb2RlbF92YXJpYWJsZXMsIAogICAgICAgICAgIGZhbWlseSA9IGJpbm9taWFsKQoKbW9kZWxfdmFyaWFibGVzICU+JSAKICBtb2RlbHI6OmFkZF9wcmVkaWN0aW9ucyhpbml0LCB0eXBlID0gJ3Jlc3BvbnNlJykgJT4lIAogIHBsb3RfbHkoeCA9IH5hZ2UsIHkgPSB+IHByZWQpICU+JSAKICBhZGRfbGluZXMoY29sb3IgPX4gbGlidXNlciwgbGluZSA9IGxpc3Qoc2hhcGUgPSAic3BsaW5lIikpICU+JSAKICBsYXlvdXQodGl0bGUgPSAnUHJlZGljdGVkIFByb2IuIEF3YXJkID4gMSBtaWwnKQpgYGAKCgoKIyMgUGxvdGx5CgpVc2UgZ2dwbG90bHkgdG8gdHVybiBvdXIgZm9ybWVybHkgc3RhdGljIHBsb3RzIGludG8gaW50ZXJhY3RpdmUgb25lcy4gCgpgYGB7ciBwbG90bHkzfQpwID0gbW9kZWxfdmFyaWFibGVzICU+JSAKICBnZ3Bsb3QoKSArCiAgZ2VvbV9kZW5zaXR5KGFlcyh4ID0gbG9nKGF3YXJkX3RvdGFsX2Ftb3VudCksIAogICAgICAgICAgICAgICAgICAgY29sb3IgPSBsaWJ1c2VyLAogICAgICAgICAgICAgICAgICAgZmlsbCA9IGxpYnVzZXIpLAogICAgICAgICAgICAgICBhbHBoYSA9IC4yKSArIAogIGZhY2V0X3dyYXAofiBnZW5kZXIpIApnZ3Bsb3RseSgpCmBgYAoKCgoKIyMgUHl0aG9uIGV4YW1wbGVzCgojIyMgSW5pdAoKYGBge3B5dGhvbiBweV9pbml0LCBlbmdpbmUucGF0aD0gJy9Vc2Vycy9taWNsL2FuYWNvbmRhMy9iaW4vcHl0aG9uJ30KaW1wb3J0IHBhbmRhcyBhcyBwZAppbXBvcnQgbnVtcHkgYXMgbnAKCmRlbW9ncmFwaGljcyA9IHBkLnJlYWRfY3N2KCdkYXRhL2RlbW9zX2Fub255bWl6ZWQuY3N2JykKaWRzID0gcGQucmVhZF9jc3YoJ2RhdGEvaWRzX2Fub255bWl6ZWQuY3N2JykKbW9kZWxfdmFyaWFibGVzID0gcGQucmVhZF9jc3YoJ2RhdGEvbW9kZWxfdmFyaWFibGVzX2Fub255bWl6ZWQuY3N2JykKYGBgCgoKIyMjICBHcm91cGluZyBhbmQgU3VtbWFyaXppbmcgZGF0YQoKYGBge3B5dGhvbiBweV9zdW1tYXJ5fQpkZW1vZ3JhcGhpY3MuZGVzY3JpYmUoaW5jbHVkZSA9ICdhbGwnKQpgYGAKYGBge3B5dGhvbiBweV9zdW1tYXJ5X251bWVyaWN9CmRlbW9ncmFwaGljcy5kZXNjcmliZShpbmNsdWRlID0gW25wLm51bWJlcl0pCmBgYAoKYGBge3B5dGhvbiBweV9zdW1tYXJ5X3N0cmluZ30KZGVtb2dyYXBoaWNzLmRlc2NyaWJlKGluY2x1ZGUgPSBbbnAub2JqZWN0XSkKYGBgCgoKCgpgYGB7cHl0aG9uIHB5X2dyb3VwX2J5LCBldmFsPVRSVUV9CmxpYl9ncm91cCA9IGRlbW9ncmFwaGljcy5ncm91cGJ5KCdsaWJ1c2VyJywgc29ydD1UcnVlLCApCgojIGF1dG9tYXRpY2FsbHkgY2hvb3NlcyBudW1lcmljCmxpYl9ncm91cC5tZWFuKCkKYGBgCgoKYGBge3B5dGhvbiBweV9ncm91cF9ieTIsIGV2YWw9VFJVRX0KbGliX2dyb3VwLmdldF9ncm91cCgwKS5oZWFkKCkKYGBgCgoKYGBge3B5dGhvbiBweV9ncm91cF9ieTMsIGV2YWw9VFJVRX0KbGliX2dyb3VwLnNpemUoKQpgYGAKCmBgYHtweXRob24gcHlfZ3JvdXBfYnk0LCBldmFsPVRSVUV9CmxpYl9ncm91cC5kZXNjcmliZSgpCmBgYAojIyMgTWFwcGluZyBhIGZ1bmN0aW9uCgpgYGB7cHl0aG9uIHB5X21hcH0KeCA9IFtbMSwyLDNdLCBbNCw1LDZdLCBbNyw4LDldXQpsaXN0KG1hcChucC5zdW0sIHgpKQpgYGAKCmBgYHtweXRob24gcHlfbWFwMn0KZGVtb2dyYXBoaWNzLnNlbGVjdF9kdHlwZXMoJ251bWJlcicpLmFwcGx5KG5wLm1lYW4pCmBgYAoKCiMjIyBWaXN1YWxpemF0aW9uCgpgbWF0cGxvdGxpYmAgaXMgdGhlIG1vc3QgY29tbW9uIHZpc3VhbGl6YXRpb24gbW9kdWxlIGluIFB5dGhvbiwgdGhvdWdoIGl0J3MgZmFpcmx5IGRhdGVkIGF0IHRoaXMgcG9pbnQuIEFzIHN1Y2ggd2UnbGwgdXNlIGEgZ2dwbG90IGltcGxlbWVudGF0aW9uIGluIHB5dGhvbiBjYWxsZWQgYHBsb3RuaW5lYC4KClVuZm9ydHVuYXRlbHkgZm9yIHBsb3RseSwgdGhlIGludGVyYWN0aXZpdHkgbWFrZXMgaXQgdW51c2FibGUgd2l0aGluIHRoZSBSIG5vdGVib29rIChhdCBwcmVzZW50KSwgc28geW91IG1heSBuZWVkIHRvIHN3aXRjaCB0byBBbmFjb25kYSBvciBvdGhlciBJREUgdG8gdHJ5IG90aGVyIG1vZHVsZXMgbGlrZSBgcGxvdGx5YC4gIEV2ZW4gUHl0aG9uIHVzZXJzIHdpbGwgc3RpbGwgdXNlIFIgZm9yIGVhc2llciB2aXN1YWxpemF0aW9uIHRob3VnaCwgc28gZmVlbCBmcmVlIHRvIGRvIHdoYXQgeW91IGxpa2UgdGhlcmUsIHRoZW4gdXNlIGdncGxvdCBldGMuIGluIFIgd2hlbiB0aGUgdGltZSBjb21lcy4KClRoYXQgc2FpZCwgSSdsbCBzaG93IGEgY291cGxlIHBsb3RzCgpgYGB7cHl0aG9uIHBsb3RuaW5lfQppbXBvcnQgcGxvdG5pbmUKYGBgCgpgYGB7cHl0aG9uIGdnMX0KZHBsb3QgPSBkZW1vZ3JhcGhpY3NbKGRlbW9ncmFwaGljcy5hd2FyZF95ZWFyX3N0YXJ0IDwgMjAyMCkgJiAoZGVtb2dyYXBoaWNzLmF3YXJkX3llYXJfc3RhcnQgPiAxOTkwKSAmIChkZW1vZ3JhcGhpY3MuYXdhcmRfdG90YWxfbG9nID4gMTEpXQpkcGxvdApnZ3Bsb3QoZHBsb3QsIGFlcyh4PSdhd2FyZF95ZWFyX3N0YXJ0JywgeT0nYXdhcmRfdG90YWxfbG9nJykpICsgZ2VvbV9zbW9vdGgoYWVzKGdyb3VwID0gJ2xpYnVzZXInLCBjb2xvciA9ICdsaWJ1c2VyJykpCmBgYAoKCmBgYHtweXRob24gZ2cyfQp5ZWFyX2F3YXJkX2F2ZXJhZ2UgPSBkZW1vZ3JhcGhpY3NbZGVtb2dyYXBoaWNzLmxpYnVzZXIgPT0gMV0uZ3JvdXBieShbJ2F3YXJkX2hvbWVfZGVwdCcsICdhd2FyZF95ZWFyX3N0YXJ0J10pLm1lYW4oKQp5ZWFyX2F3YXJkX2F2ZXJhZ2UKYGBgCgoKCmBgYHtweXRob24gcHlfcGxvdGx5X2luaXR9CmltcG9ydCBwbG90bHkKaW1wb3J0IHBsb3RseS5wbG90bHkgYXMgcHkKZnJvbSBwbG90bHkub2ZmbGluZSBpbXBvcnQgaW5pdF9ub3RlYm9va19tb2RlCmltcG9ydCBwbG90bHkuZ3JhcGhfb2JqcyBhcyBnbwpwbG90bHkub2ZmbGluZS5pbml0X25vdGVib29rX21vZGUoY29ubmVjdGVkPVRydWUpCmBgYAoKCmBgYHtweXRob24gcHlfcGxvdGx5fQp5MCA9IG5wLnJhbmRvbS5yYW5kbig1MCktMQp5MSA9IG5wLnJhbmRvbS5yYW5kbig1MCkrMQoKdHJhY2UwID0gZ28uQm94KAogICAgeT15MAopCnRyYWNlMSA9IGdvLkJveCgKICAgIHk9eTEKKQpkYXRhID0gW3RyYWNlMCwgdHJhY2UxXQpweS5pcGxvdChkYXRhKQpgYGAKCgpgYGB7cHl0aG9ufQpgYGAK